Traditional r-adaptive mesh refinement aims to reduce a priori error estimates within a fixed computational budget by applying an optimal mesh deformation map. This results in a slow process due to the need to solve a nonlinear meshing PDE and only minimizes the a priori error estimate. Building on recent advances in GNNs inspired by PDEs, we use Graph Neural Diffusion (GRAND) to learn the deformation map as a discretization of a continuous non-homogeneous diffusion process. This strong inductive bias leads to a rapid and robust graph-based AMR approach with provable guarantees of mesh quality. Using a discrete adjoint method related to PDE-constrained optimization, the GNN is trained to directly minimize the local truncation error, achieving significant acceleration and error reduction.